A generally known method of observing mixed reality space includes the video see-through method and the optical see-through method. With the former, image data of actual space sensed by a video camera or the like is synthesized with an image such as computer graphics (CG) or the like and the synthesized image is exhibited to an observer. With the latter, an image such as CG is overlapped on a half-transmitting (see-through) display device.
There are three methods for exhibiting mixed reality space to an observer: having the observer wear a head mounted display (HMD) for exhibiting an image, setting a display device near the observer to exhibit an image, and having the observer operate a hand-held display device and changing the displayed image based on motion of the observer's body for exhibiting an image. As an apparatus for expressing mixed reality space, a combination of the three exhibiting methods and the aforementioned video see-through method or the optical see-through method may be considered.
There are cases that the mixed reality space, realized by any of the aforementioned combination, is shared by plural persons for performing cooperative work. In this case, grasping where each person is looking at in the mixed reality space contributes to smooth proceeding of work. For instance, assume that a worker (exhibiter) wants to show an object to another worker (observer). If the exhibiter is aware of the visual field of the observer, the exhibiter can show the object to the observer for certain. In addition, an effective way of exhibiting, e.g., a part of the object to be emphasized is directed to the observer when the object is shown, can easily be realized. If the observer is an elderly who is difficult to change the visual field by himself/herself, the exhibiter who is exhibiting the object can carry the object to the observer's visual field and show the object to the observer. To date, however, in order to grasp the observer's visual field, it is necessary for the exhibiter to predict the observer's visual field by visually identifying the external state of the observer and the position/orientation of an image sensing apparatus or the like used in observation.
According to the disclosure of Japanese Patent Application Laid-Open (KOKAI) No. 2000-353248, when a HMD for displaying mixed reality space is worn, the direction of observer's visual line is detected and indicated in the image of the exhibiter's mixed reality space. More specifically, a virtual object representing the direction of the observer's visual line is superimposed at the location of the observer's head wearing the HMD in the image which is sensed from the exhibiter's view point in the direction of the observer's existence.
However, the information obtained by the method disclosed in Japanese Patent Application Laid-Open (KOKAI) No. 2000-353248 is only the data regarding the direction of visual line, and does not clearly tells the exhibiter the visual field of the observer.
Furthermore, there is another problem in the system which performs clipping in a rendered virtual object. Clipping is performed in the following case. For instance, if an object is exhibited off the area between the front clipping surface 302 and the rear clipping surface 303 of the view volume 101 shown in FIG. 3, virtual data of the exhibited object is not displayed (rendered) even if the object is within the visual field. In this system, if the object is exhibited, for instance, behind the rear clipping surface 303, virtual data of this object is not displayed even though the object exists within the observer's visual field. However, the exhibiter is unable to determine such situation based only on the indication of the aforementioned virtual object which represents the direction of the observer's visual line.